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ABSTRACT 


The accuracy of the discrete reliability growth model developed bv Armv Material 
System Analysis Activity (AMSAA) ıs analysed. The mean, standard deviation, and 
95 percent confidence interval of the estimate of reliability resulting from simulating the 
AMSAA discrete reliability growth model are computed. The mean of the estimate of 
reliability from the AMSAA discrete reliability growth model is compared with the mean 
of the reliability estimate using the Exponential discrete reliability growth model devel- 
oped at the Naval Postgraduate School and with the actual reliability which was used 
to generate test data for the replications in the simulations. The testing plan simulated 
in this study assuines that the mission tests (go-no-go) are performed until a predeter- 
mined number of failures occur at which time a modification is made. The main results 
are that the AMSAA discrete reliability growth model always performs well with con- 
cave growth patterns and has difficulty in tracking the actual reliability which has con- 
vex growth pattern or constant growth pattern when the number of failures specified 


egal to one. 
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I. INTRODUCTION 


The test-analyze-and-fix scenario is frequently followed in order to achieve high re- 
liability under current DOD design and development policies during early development. 
tem will Usually be tested Until it fails, The failure is analvzed to determine its 
cause, and what needs to be done to remove the cause of failure. Appropriate changes 
are made and more items are tested until the next failure occurs. After each modifica- 
MT io [he item, it has a new reliability and after the A™ mouilication we are in the A” 
reliability growth phase and all items tested in this phase have common reliability R, 
This procedure is repeated several times until the requirement for reliability ts achieved. 
Through this procedure a reliability growth pattern is established. Reliability growth 
models have been developed to estimate reliability from phase to phase for this type of 
test program. One such model is the Army Material System Analysis Activity 
(AMSAA) Discrete Reliability Growth Model. 

The purpose of this paper is to perform an accuracy analysis of the AMSAA discrete 
reliability growth model. Performance evaluation of the AMSAA discrete reliability 
growth model was done using monte carlo simulation to generate test data which in turn 
Was used to exercise the AMSAA computer program to compute the estimate of the 
reliability for each phase. The reliability estimates obtained from the AMSAA model 
are compared with the actual reliability in a predeternuned sequence of reliabilities which 
used to generate test data. In addition these values are compared with the reliability 
estimate obtained from the Exponential discrete reliability growth model which has been 
analvzed at the Naval Postgraduate School [Ref. I, 2, and 3]. General description of the 
analysis used in this paper is described below : 

For each phase, 
e Assign value R,, the reliability for i phase 
e Specify F,, the number of failures specify to stop the phase 


e Generate N,, the number of tests needed to obtained F failures 


“NG ollect the test data, N and E, 


* Compute A, the estimate of R, 


e Replicate this scenario 500 times 


* Compute the sample mean R, and sample standard deviation Sa 


Compute a 95% confidence interval for E[R,] 
Compare R, with R, in graphical form 


Compare R, with the estimate of reliability using the Exponential discrete reliability 
growth model with the same data 


Prepare appropriate graphs. 


H. AMSAA DISCRETE RELIABILITY GROWTH MODEL 


The AMSAA discrete reliability growth model [Ref. 4] was developed by L.Crow in 
[983 1s based on a learning curve approach that had been mentioned by Duane [Ref. 
5] This model is applicable when items under test are scored as success or failure. 
The model is derived from a reliability growth model for continuous data , 1.c., time to 


failure data. [t is based on the Duane reliability growth concepts. 


A. INTERPRETATION OF LEARNING CURVE PROPERTY 
BetC@nmebe tile CUImMUlative (alllre Tater nN (t) oe the total number of failures by time 


ee ere tis tie cumulative test ume. The ratio of K(t) and t is equal to C(t) 


A(t) 
[ 





= C(t) 


Duane observed that In C(t) was linear when plotted against In t 


That is, 


In C(r) 2 ó—a Int 


Duane expressed this relationship differently as follows : 














Dez 
In m ó—alnt 
KL l 
ln a 
[ 
K(t 
In (2 = O 
AO) o, 
pn = € 
let l1-a=8 
and let ef=} 
Eis As 


Consequently, if In C(t) is a linear function of In t, this implies KL = 44º, which is the 


learning curve property for K(t). (Ref. 4 page I]. 


B. AMSAA DISCRETE RELIABILITY GROWTH MODEL DEVELOPMENT 
The discrete reliabilitv growth model developed at AMSAA uses attributes data. 

This model is described as follows : 

As Number of ss or one EN 

T, 2 Cumulative number of trials through configuration ; 

i 

T NE 

In general : 

T=N+M+N+..+N 

M, 2 Number of failures for configuration 1 

K, 2 Cumulative number of failures through configuration 1 

KR, aM 

Ro = MM, NO 

n Fee 

K,— M, M, My, ++.. +M 

E[ AJ = Expeeted sale of A,. 
The model assumes that log E[K] is linear when plotted against log 7;. This implies 
ELK] 2 AT? . [Ref. 4 page I to 4]. 

ELA] 2AT? z BN, 


iT 
Eg 


«VI 
ELK] =R} = Pay, t PW, 
i ae Pe 


ESOS = A T?—} e 


¿MAT 
is N, 
In general : 
TX 
Vai NY 


[Ref. 4 page 5 to 6). 


C. ESTIMATION PROCEDURE FOR AMSAA DISCRETE MODEL 
The maximum likelihood estimates 4 and ß for 4 and f. were derived bv AMSAA as 
follows : 
Ler, 
P = Probability of failure for configuration i 
P, = Estimated failure probability for configuration i 
R,= Reliability for configuration 1 


R, = Estimated reliability for configuration i 


Then 

P B AT? — AT? , 
I N, 

= |-P 


where maximum likelihood estimates of A and f are values satisfying : 


k 
Nim Mi; 0 : . p 
ni ts ——— ene cen Ted ssl 
Eat [A en; TE O eel ie] 


Poe 
be 


and, 


Sc 


^| M, dE 


———— 2 ———— ——— If - TÉ.) 2 0. 
ET AA BE 


[Ref. 4 page 10 to 12]. 


Poe 


In this paper À and f were computed using the AMSAA computer program. 


HI. EXPONENTIAL DISCRETE RELIABILITY GROWTH MODEL 


The Exponential discrete reliabilitv growth model has been analyzed at the Naval 
Postgraduate Schoo! in two theses [Ref. L, 2], and by Corcoran and Read |Ret. 3] ieee 
Corcoran and Read have compared several popular reliability growth models. — This 
model serves as a model comparison to the AMSAA discrete reliability growth model. 
The Exponential discrete reliability growth model uses only attribute data. It does not 
require any assumption about the distribution of the time to failure. This model is de- 
scribed briefly as follows : 

ou: 
K,= The rehabilitv of the component in phase / 
R,= l — exp{ — (a + pÒ? where 1=0 


¡=0 means the phase prior to any modification 


The parameter estimates «, and f, ofa and £ for phase i are computed using linear 
regression methods and an unbiased estimator for (a + fii) 
F =the total number of failure during phase 


NV, =the number of tests between the (j — 1)* failure and J^ failure, including these 
in phase / 


O E 


Y,,= unbiased estimator of (a 4- Di) using j^ sequence test. in phase i 


An unbiased estimator Y,, for (a + fi) [ChernofT and Woods 1965] is known to be : 





0 if Ni=l 
r= (a + Bi) =| I l 
—— Je aes r oos 
1+ eae Nol if Nie 
fori =Oel, 2... sand Jd M 
Since Npe s Nae oee Vp, are independent random variables, then: 


ne aj Y, uc Y.) : l ; 
Y= A A SU also an unbiased estimates for (a + fi) . 


l 


The least square estimates x, and f, for « and f at phase i are: 


p === for i=1, 2, 3, 
Var: 
La 
Ji 
and, 
a= Y =P for i=1,2,3, 
where : 
Ben 
B (i+ 1) 
2001224 HY 
E TEST 


By using « and f, the estimate of reliability for every phase í can be computed as fol- 


IONS : 


a 


FED ENO (Gb for i=1,2,3,.. 
The estimate of reliability for the original version of the component R, Is given by : 
HE IE EXPO EF). 


(Ref. 6 page 3-1 to 3-3]. E 

In this paper the value of the mean regression estimate R, of reliability and the value 
of standard deviation of the estimate of reliability Sa were obtained from a computer 
program used in J. Chandler thesis [Ref. 2]. The equations for computing the reliability 


growth values R, are easily solved using a hand-held calculator. 


IV. MONTE CARLO SIMULATIONS 


Since the AMSAA model is interested in the number of trials until the x^ failure 
occurs, the Monte Carlo simulation generates random variable using an algorithms de- 
veloped by Fishman 1978 [Ref. 7] , and a subroutine from The New Naval Postgraduate 
School Random Number Package LLRANDOMII 1981 [Ref. 8] as a random number 
generator for real uniform from O to |. Given p, the probability of failure) and ra 
number of failure for every phase, the computer simulation generated the number 
trials until the na failure. Specifically let X be the random variable of interest; tema 
ber of trials until the r^ failure, then X is called negative binonual random variable with 


parameter rand p. The probability function for .X is, 
PA) = (ED Dar k=rr+lr+2.. 1>0 


The Algorithms for Computer Simulation : 
l. let 4 and B be double precision variables 
2. w=(p) 
lfr<0, (1=p)<0, (l—=p)21, w<0, w>1l goto9 
d dear, A=w, B=w and 0 =(l — plr — 1) 
. Generate U, uniform random number from 0 to | 
. ITU<A or 420.999999 or B«0.000001, go to IO 
+= X LL B=B(0/X+(1-p)) and A=A+B 
S. Go to Jd 


9. Print error message and stop 


t2 


Gy can 


=N 


JO Continue. 
[Ref. 7 page 354]. 


V. TEST PROCEDURE 


The AMSAA model is evaluated using eight different sets of reliabilitv values (the 
actual growth pattern) and two different sets of inputs of number of failure per phase. 
This gives a total of sixteen cases. Table 1 describes all 16 cases. The set of reliability 
Malues tor case ¡1s the same tnat foncase +8, i=1,2,..., 8. For cases À through 
S the number of failures per phase are equal to one and for cases 9 through 16 the 
number of failures per pirase are equal to three. The diagram in Figure 1 summarizes 


the simulation procedure and the consequent analysis. 


Table l. ACTUAL RELIABILITY FOR 16 CASES 


CASE NUMBERS 
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THE EXPONENTIAL COMPUTER PROGRAM 

Computes: 

- The mean regression estimate of 
reliability 

- The standard devtation of regreasion 

astimate of reliability 







INPUT DATA 


- Number of failure for every phase | 
- Probability of failure for avery phase | 








MONTE CARLO SIMULATION 
Generates the Negative Binomial RLV. 












THE AMSAA COMPUTER PROGRAM 

Computes: 

- The estimate of reliability 

- The mean of the estimate of reliability 

- The standard deviation of the estimate 
ot reliability 


- 95% Cl of the mean of the estimate of 
reliability 








PERFORMANCE PLOT 
- Plots of the mean of the estimate ot 
reltability from both model and the 

actual reliability 
- Plots of the standard deviation of 
the estimate of reliability from both 
model 

















Figure {. Block diagram of the analysis 


VI. ANALYSIS PROCEDURE 


A. ACCURACY 

Figures 4 through 35 in the Appendix provide a visual display of the AMSAA dis- 
crete reliability growth model accuracy bv comparing the growth line for the AMSAA 
ne actual rehabiity growth pattern. These graphs atso provide plots for the Ex- 
ponential discrete reliabilitv growth model using the same mput data as that used in the 


AMSAA discrete rettabilitv growth model. 


B. VARIABILITY 
In addition to the tracking ability of the reliability point estimates R,, the user is 
also interested in the variability of R. Five hundred replications were run for each of 


the 16 cases and each of the 10 phases, this provided : 





SOO 
ee | . 
¿300 > R 
E 
and 
500 sa 
ae 
E 
E 499 
A  rforeach of the 16 cases. . 


The algorithm used to compute the mean and standard deviation is developed by 
Miller 1982 (Ref. 9 page 17 to 19]. Standard deviation of the reliability estimates from 


both the AMSAA and the Exponential model are plotted. 


C. CONFIDENCE INTERVAL 
A 95% two sided confidence interval for ETR] 1s computed for each model for all 


The equation used for these confidence limits are as follows : 


16 cases. 
i (ESS 
Lre o 
. 500 
ande 


- (1.96)55 
L, =k -——* 
x 500 


for i=1,2,... 10or each ok lo cases: 


VH. RESULTS AND CONCLUSIONS 


The resuits for all 16 cases can be seen in the Appendix. The case number appears 
at the table caption or at the figure caption. All of the results for each case are divided 
into two categories, 1.c., the tabulated statistic and performance plot. In this chapter 
ieee. ure 2 and [ruie 3 are explained as an example of the result from case 1 


Sr data set. 


A. TABULATED STATISTICS 

Table 2 indicates that testing was done until one failure occured after which a 
change in the item was made. The actual reliabilitv growth values for each of the 10 
phases was constant at 0.60043, 1.e., no growth actually occured. It is important to 
simulate this case in order to examine the ability of the growth model to detect no 
growth. Some reliability growth models have a built in assumption that some growth 
always takes place after a design change. 

The values of R, for /2 1,2, ...,I0 are given in column 4 for the AMSAA model 


and in column $ for the Exponential model, thus for phase 7, the AMSAA model vielded 


z= 
` 


R,=0.580187 and the Exponential model yielded a value of 0.525144. The 
coresponding values of the standard deviation are 0.124669 and 0.261854 for the 


AMSAA and the Exponential model respectively. 


B. PERFORMANCE PLOT 
Figure | is a plot of R versus í for the AMSAA and Exponential models. It also 
displays a plot of the actual reliabilities R,. Figure 2 is a plot of standard deviation for 


case | for both the AMSAA and the Exponential model. 


Table 2. STATISTICS FOR CASE I 
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The reliability growth pattern comparison plot case 1 
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The standard deviation comparison plot case 1 


C. SUMMARY AND CONCLUSIONS 
To analyze the test results for cases 1 through 16, all were divided into categories, 
Le., constant growth pattern, concave with rapid growth pattern, concave and convex 


growth pattern, convex growth pattern (see Appendix). 


l. Constant Growth Pattern 

The AMSAA model didn't track the actual reliability too well for cases 1, 2, and 
3 (the number of failure per phase was set equal to one). The AMSAA developed a 
concave growth pattern, eventhough in these cases the actual reliability was constant. 
Furthermore for case 3 the AMSAA model performance became worse since it had de- 
creasing pattern and went below the actual reliability at phase 10. However when the 
number of failure increased to three, the AMSAA model tracked the actual reliability 
quite Well. The mean of the estimate of reliability was close to the actual reliability, and 


the standard deviation of the estimate of reliability was very small. 


2. Concave with Rapid Growth Pattern 
This tvpe of actual reliability growth pattern is represented in cases 4, 5, 12, and 
13. The AMSAA model performed well in tracking actual reliability growth, especially 
for case 4, case 5, and case 13, where it 1s close to the actual reliability with very small 
standard deviation of the estimate of reliability. For case I2, the AMSAA model for 
some reason could not track the actual reliability very well. It performed alinost con- 
stant growth, with a small decrease out through phase 10. This is a strange phenomena. 


This case was run several times with the same result. 


3. Concave and Convex Growth Pattern 
The AMSAA model has a problem tracking reliability growth pattern estab- 
lished in cases 6, 7, 14, and 15. The AMSAA model seems to display a concave growth 
pattern, it could not track the actual reliability which has a concave followed by a con- 
vex growth pattern. This is probably because the cumulative assumption inherent in 
the AMSAA model does not work well when the reliability growth has a convex growth 


DUUM 


16 


4. Convex Growth Pattern 
The AMSAA model also had difficulty in tracking the actual reliability growth 
pattern for cases 8 and 16, it just performed concave growth pattern for both cases. 


The standard deviation of the estimate of reliability was good in these cases. 


5. Summary 
It appears that the AMSAA discrete reliability growth model ts more appropri- 
ate for reliability growth pattern that has the following characteristics : 
1. Concave with rapid growth pattern 


2. Constant growth pattern with number of failure specified more than once. 


lt appears that the user should be wary of using the AMSAA discrete reliability 
growth model when the actual reliability growth has the following characteristics : 
I. Constant growth pattern with number of failures specified equal to one 
pee oncave growth followed by convex growth pattern 


Extonvex growth pattern. 


Also the user should note that other discrete reliabilitv growth models are available 
which for some growth pattern performed better than the AMSAA model and which can 


be programmed on a hand-held calculator. 
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Figure 4. The reliability growth pattern comparison plot case 1 
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Figure 5. The standard deviation comparison plot case 1 
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Table 4. STATISTICS FOR CASE 2 
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The reliability growth pattern comparison plot case 2 
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Table 6. STATISTICS FOR CASE 4 
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Figure 10. The reliability growth pattern comparison plot case 4 
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Table 7. STATISTICS FOR CASE 5 
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Figure 12. The reliability growth pattern comparison plot case 5 
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Figure 16. The reliability growth pattern comparison plot case 7 
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Figure 17. The standard deviation comparison plot case 7 
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Table 10. STATISTICS FOR CASE 8 
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Figure 18. The reliability growth pattern comparison plot case 8 
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Figure 19. The standard deviation comparison plot case 8 
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Figure 20. The reliability growth pattern comparison plot case 9 
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Figure 21. The standard deviation comparison plot case 9 
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Table 12. STATISTICS FOR CASE 10 
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Figure 22. The reliability growth pattern comparison plot case 10 
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Figure 23. The standard deviation comparison plot case 10 
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Figure 24. The reliability growth pattern comparison plot case 11 


STANDARD DEMATION 


1.0 


0.6 


0.6 


V = AMSAA 
O = EXPONENTIAL 


RELIABILITY 
0 





Figure 25. The standard deviation comparison plot case 11 
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Table 14. STATISTICS FOR CASE 12 
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Figure 26. The reliability growth pattern comparison plot case 12 
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Figure 27. The standard deviation comparison plot case 12 
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Table 15. STATISTICS FOR CASE 13 
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Figure 28. Ihe reliability growth pattern comparison plot case 13 
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Figure 29. The standard deviation comparison plot case 13 
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Table 16. STATISTICS FOR CASE 14 
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Table 17. STATISTICS FOR CASE 15 
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Figure 32. The reliability growtli pattern comparison plot case 15 
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Figure 33. The standard deviation comparison plot case 15 
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Table 18. STATISTICS FOR CASE 16 
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Figure 34. The reliability growth pattern comparison plot case 16 
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Figure 35. The standard deviation comparison plot case 16 
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